Skip to content

Use check-run-results for run-fail test stderr #133460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Use check-run-results for run-fail test stderr #133460

wants to merge 1 commit into from

Conversation

clubby789
Copy link
Contributor

Fixes #65865

@rustbot
Copy link
Collaborator

rustbot commented Nov 25, 2024

r? @estebank

rustbot has assigned @estebank.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc PG-exploit-mitigations Project group: Exploit mitigations S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Nov 25, 2024
clubby789
clubby789 commented Nov 25, 2024
View reviewed changes
@clubby789 clubby789 added C-cleanup Category: PRs that clean code up or issues documenting cleanup. and removed PG-exploit-mitigations Project group: Exploit mitigations labels Nov 25, 2024
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@clubby789
Copy link
Contributor Author

oops, non anonymized panic strings

@clubby789 clubby789 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 25, 2024
@rustbot rustbot added the PG-exploit-mitigations Project group: Exploit mitigations label Nov 25, 2024
@rustbot
Copy link
Collaborator

rustbot commented Nov 25, 2024

Some changes occurred in tests/ui/sanitizer

cc @rust-lang/project-exploit-mitigations, @rcvalle

@clubby789
Copy link
Contributor Author

Reverted a few tests that have more variable output

@rustbot rustbot added the A-compiletest Area: The compiletest test runner label Nov 25, 2024
@rustbot
Copy link
Collaborator

rustbot commented Nov 25, 2024

Some changes occurred in src/tools/compiletest

cc @jieyouxu

@clubby789 clubby789 force-pushed the run-fail-output branch 2 times, most recently from a992f44 to abcff47 Compare November 25, 2024 17:54
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@clubby789
Copy link
Contributor Author

Hmm, we can't normalize $CARGO_HOME naively like other paths, because we have tests with paths containing .../cargo... - and /cargo is the CI $CARGO_HOME

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
Zalathar added a commit to Zalathar/rust that referenced this pull request Dec 29, 2024
@Zalathar
Rollup merge of rust-lang#133460 - clubby789:run-fail-output, r=estebank
8ec576c 
Use `check-run-results` for `run-fail` test stderr

Fixes rust-lang#65865
@Zalathar Zalathar mentioned this pull request Dec 29, 2024
Closed
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 29, 2024
@bors
Auto merge of rust-lang#134878 - Zalathar:rollup-ebqmxw7, r=Zalathar
4a7bd52 
Rollup of 4 pull requests

Successful merges:

 - rust-lang#122565 (Try to write the panic message with a single `write_all` call)
 - rust-lang#133460 (Use `check-run-results` for `run-fail` test stderr)
 - rust-lang#134627 (Avoid ICE in borrowck)
 - rust-lang#134799 (nits: Cleanups in `librustdoc::clean`)

r? `@ghost`
`@rustbot` modify labels: rollup
Zalathar added a commit to Zalathar/rust that referenced this pull request Dec 29, 2024
@Zalathar
Rollup merge of rust-lang#133460 - clubby789:run-fail-output, r=estebank
67e7888 
Use `check-run-results` for `run-fail` test stderr

Fixes rust-lang#65865
@Zalathar Zalathar mentioned this pull request Dec 29, 2024
Closed
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 29, 2024
@bors
Auto merge of rust-lang#134879 - Zalathar:rollup-olayokx, r=Zalathar
869b3cc 
Rollup of 3 pull requests

Successful merges:

 - rust-lang#133460 (Use `check-run-results` for `run-fail` test stderr)
 - rust-lang#134627 (Avoid ICE in borrowck)
 - rust-lang#134799 (nits: Cleanups in `librustdoc::clean`)

r? `@ghost`
`@rustbot` modify labels: rollup
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 29, 2024
@bors
Auto merge of rust-lang#133460 - clubby789:run-fail-output, r=estebank
2a2f01f 
Use `check-run-results` for `run-fail` test stderr

Fixes rust-lang#65865
@bors
Copy link
Collaborator

bors commented Dec 29, 2024

⌛ Testing commit e1a9aff with merge 2a2f01f...

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@bors
Copy link
Collaborator

bors commented Dec 29, 2024

💔 Test failed - checks-actions

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 29, 2024
@clubby789
Copy link
Contributor Author

Looks spurious

@bors retry

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 29, 2024
Zalathar added a commit to Zalathar/rust that referenced this pull request Dec 29, 2024
@Zalathar
Rollup merge of rust-lang#133460 - clubby789:run-fail-output, r=estebank
10c8c3e 
Use `check-run-results` for `run-fail` test stderr

Fixes rust-lang#65865
@Zalathar Zalathar mentioned this pull request Dec 29, 2024
Closed
@clubby789
Copy link
Contributor Author

@bors r-
Misread the error, sanitizer tests are fragile due to addresses and should not have stderr checked

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 29, 2024
@jieyouxu
Copy link
Member

jieyouxu commented Dec 29, 2024 •
edited
Loading

Also leaving a comment here so discussions don't get lost elsewhere:

We should be a bit careful with replacing //@ error-pattern with //@ check-run-results, because they are not mutually exclusive and they aren't really substitutes (if anything, they are complementary):

  • //@ error-pattern will look for a regex match in the run stderr (and stdout combined, IIRC) if the test is //@ run-fail.
  • //@ check-run-results instructs compiletest to snapshot run stderr/stdout, and compare a run's stderr/stdout against those exactly.

Concretely, this means:

  • //@ check-run-results is for when you really want the exact run stderr/stdout to not change. This will be especially relevant to tests that may include things that change between runs.
    • If a test needs such exact snapshot comparisons, you can still have error-pattern to enforce the check on the pattern that is critical, to prevent you from accidentally blessing the important thing away.
  • //@ error-pattern is good for checking the critical parts of the run stderr/stdout, cannot be easily blessed away, and is typically less sensitive than full stderr/stdout comparisons.

@clubby789
Copy link
Contributor Author

Going to close; I think a blanket approach is unhelpful as it's going to be flaky (especially when stderr contains line numbers from external crates or stdlib), and doesn't provide much value

@clubby789 clubby789 closed this Dec 29, 2024
@clubby789 clubby789 deleted the run-fail-output branch December 29, 2024 11:52
@clubby789 clubby789 mentioned this pull request Dec 29, 2024
Closed
3 tasks
@jieyouxu
Copy link
Member

jieyouxu commented Dec 29, 2024 •
edited
Loading

cc Audit {check-run-results, error-pattern, check-stdout, normalize-*, dont-check-compiler-*, run-rustfix, rustfix-only-machine-applicable, forbid-output} family of directives.

@jieyouxu
Copy link
Member

cc Audit run-fail ui tests to see if they need exact run stderr/stdout #134889.

@jieyouxu jieyouxu mentioned this pull request Jul 8, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@estebank estebank

Labels
A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc C-cleanup Category: PRs that clean code up or issues documenting cleanup. PG-exploit-mitigations Project group: Exploit mitigations S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cleanup: Migrate run-fail tests in src/test/ui from error-pattern to check-run-results
6 participants
@clubby789 @rustbot @rust-log-analyzer @bors @estebank @jieyouxu